Robotic automation processes that require two or more components to be aligned relative to each other, such as pick and place operations, material removal, and the assembly of electronic devices including portable, palm-size electronic devices, such as mobile communication devices, GPS (Global Positioning System) navigation devices, and PDAs (personal data assistants) is predominantly a repetitive manual task. In particular, with regard to pick and place operations and material removal, the robotic process carrying out such operations is required to move various structural components with high precision to successfully complete the operation. Whereas, the difficulty in automating electronic device assembly is due to the unique characteristics of the small-scale or small-sized electronic components involved, which are typically in the range of 1 to 10 mm, and which must be assembled together with tight spatial tolerances. Thus, the successful automated assembly of electronic devices requires that the structural features provided by each small-scale component, such as a connection socket on a printed circuit board (PCB), be precisely aligned to allow their successful assembly.
In order to achieve precision alignment of structural components, as well as tight-tolerance, automated assembly of small-scale electronic components, vision-guided robotic alignment and assembly techniques, such as model calibration techniques and visual servoing techniques, have been utilized to compensate for position and orientation variation errors inherent in the operation of automated robotic alignment and assembly systems. Model calibration techniques rely on obtaining an accurate camera model calibration and generating an accurate robot kinematic model to achieve precise component positioning during automated robotic alignment and assembly, which is expensive, complex, and time-consuming. Visual servoing, on the other hand, requires high-speed image processing to robustly detect image error between the robot's current pose and a target robot pose at all times in order to adjust the motion of the robot so that it can close in or otherwise be adjusted toward the target pose. Unfortunately, the high-speed image processing utilized by visual servoing is complex and costly to deploy and maintain.
Due to the drawbacks of the model calibration and visual servoing techniques, camera space manipulation (CSM) control techniques were developed. Camera space manipulation uses one or more imaging cameras to image a component as it is being manipulated or otherwise moved by a robot to generate a simple local camera space model that maps the relationship between the nominal position coordinates of the robot and the appearance of structural features found on the component, such as an electrical component, that is being manipulated in camera image space. For example, the camera space model may model the appearance of structural features on a component, such as a connection socket on a printed circuit board (PCB) that is being grasped by a robotic gripper. Camera space manipulation requires only a limited number of local data points to be acquired during the positioning process to refine the local camera model in order to compensate for grasping error, robot forward kinematic error and camera model error, so as to achieve precise, tight-tolerance assembly of small-scale components, as well as precision alignment during pick and place and material removal operations for example.
However, to implement vision-guided alignment or assembly using camera space manipulation techniques, a manual setup or initialization is initially required so that the local camera space model recognizes the visual structural features provided by the component being manipulated by the robot. To complete the manual training, the visual structural features of the component, which are visually accessible to the camera, are selected by an individual on an ad-hoc basis, based on their experience. Once the features are identified, the component is brought within the field of view of the camera so that it is imaged as an image frame. In addition, the individual must manually define the center of the visual feature in the camera space field of view, while the relative position between each visual feature in robot physical space is also required to be defined. Moreover, manual initialization requires that the relative position between the visual features, which are visually accessible to the camera, and assembly features, which are outside the field of view of the camera, of the component in the image frame also be defined. However, because assembly features of the component are not visually accessible by the imaging camera, it is difficult to accurately measure the distances between visual features and assembly features manually. Unfortunately, the error resulting from the manual measurement results in an inaccurate robot camera model that is generated based on the position of the visual features. This robot camera model is used to control the robot so as to position the component so that the visually occluded assembly features are placed in the correction position. Thus, if the robot camera model is inaccurate, the robot is unable to align or assemble the components accurately with high precision, which is unwanted.
Moreover, identifying the insertion positions that identify the location of structural features where the components are to be coupled or assembled together by manually jogging or moving the robot is very difficult and requires numerous trials by a skilled robotic engineer. The initial calibration information for identifying the position of the robot and visual feature appearance data that is captured in visual camera space, is used to initialize the robot camera calibration model and is dependent on the skill of the robotic engineer who is required to establish the visibility of the visual features of the components throughout each calibration position of the camera. Unfortunately, using a manual initialization process it is difficult to ensure the reachability of the robot, ensure that the robot is able to avoid collisions with surrounding structures in the robotic cell, and to ensure that the same orientation exists between the calibration positions and the insertion positions, since each robot position may have different positioning errors caused by an imperfect robot forward kinematic model. Moreover, the manual initialization process makes it difficult to ensure the calibration positions are distributed in both camera image space and robot physical space in order to obtain desired levels of stability and sensitivity to sample noise from the camera space manipulation (CSM) control process initialization calculation, which is desired.
In addition, manual testing of the initial calibration positions is tedious, time consuming, and error prone, as this process typically requires multiple iterations of manually moving the robot in order to develop acceptable calibration positions.
Therefore, there is a need for a vision-guided alignment system that automatically initializes and configures a camera space manipulation model to compensate for position and orientation variations errors, where the visual features of the components to be assembled are automatically identified using associated CAD (computer aided design) component models. In addition, there is a need for a vision-guided alignment system that automatically initializes and configures a camera space manipulation model to automatically define the relative position between visually accessible features and the visually occluded assembly features of the components being assembled. Furthermore, there is a need for a vision-guided alignment system that automatically generates calibration positions in a computer-simulated robotic work cell to acquire data to calculate the initial camera robot calibration model, which is tested and verified in the computer-simulated robotic work cell.